Visuomotor Coordination in Reach-To-Grasp Tasks: From Humans to Humanoids and Vice Versa

Understanding the principles involved in visually-based coordinated motor control is one of the most fundamental and most intriguing research problems across a number of areas, including psychology, neuroscience, computer vision and robotics. Not very much is known regarding computational functions that the central nervous system performs in order to provide a set of requirements for visually-driven reaching and grasping. Additionally, in spite of several decades of advances in the field, the abilities of humanoids to perform similar tasks are by far modest when needed to operate in unstructured and dynamically changing environments. More specifically, our first focus is understanding the principles involved in human visuomotor coordination. Not many behavioral studies considered visuomotor coordination in natural, unrestricted, head-free movements in complex scenarios such as obstacle avoidance. To fill this gap, we provide an assessment of visuomotor coordination when humans perform prehensile tasks with obstacle avoidance, an issue that has received far less attention. Namely, we quantify the relationships between the gaze and arm-hand systems, so as to inform robotic models, and we investigate how the presence of an obstacle modulates this pattern of correlations. Second, to complement these observations, we provide a robotic model of visuomotor coordination, with and without the presence of obstacles in the workspace. The parameters of the controller are solely estimated by using the human motion capture data from our human study. This controller has a number of interesting properties. It provides an efficient way to control the gaze, arm and hand movements in a stable and coordinated manner. When facing perturbations while reaching and grasping, our controller adapts its behavior almost instantly, while preserving coordination between the gaze, arm, and hand. In the third part of the thesis, we study the neuroscientific literature of the primates. We here stress the view that the cerebellum uses the cortical reference frame representation. The cerebellum by taking into account this representation performs closed-loop programming of multi-joint movements and movement synchronization between the eye-head system, arm and hand. Based on this investigation, we propose a functional architecture of the cerebellar-cortical involvement. We derive a number of improvements of our visuomotor controller for obstacle-free reaching and grasping. Because this model is devised by carefully taking into account the neuroscientific evidence, we are able to provide a number of testable predictions about the functions of the central nervous system in visuomotor coordination. Finally, we tackle the flow of the visuomotor coordination in the direction from the arm-hand system to the visual system. We develop two models of motor-primed attention for humanoid robots. Motor-priming of attention is a mechanism that implements prioritizing of visual processing with respect to motor-relevant parts of the visual field. Recent studies in humans and monkeys have shown that visual attention supporting natural behavior is not exclusively defined in terms of visual saliency in color or texture cues, rather the reachable space and motor plans present the predominant source of this attentional modulation. Here, we show that motor-priming of visual attention can be used to efficiently distribute robot's computational resources devoted to visual processing

Learning robotic eye-arm-hand coordination from human demonstration: a coupled dynamical systems approach

We investigate the role of obstacle avoidance in visually guided reaching and grasping movements. We report on a human study in which subjects performed prehensile motion with obstacle avoidance where the position of the obstacle was systematically varied across trials. These experiments suggest that reaching with obstacle avoidance is organized in a sequential manner, where the obstacle acts as an intermediary target. Furthermore, we demonstrate that the notion of workspace travelled by the hand is embedded explicitly in a forward planning scheme, which is actively involved in detecting obstacles on the way when performing reaching. We find that the gaze proactively coordinates the pattern of eye-arm motion during obstacle avoidance. This study provides also a quantitative assessment of the coupling between the eye-arm-hand motion. We show that the coupling follows regular phase dependencies and is unaltered during obstacle avoidance. These observations provide a basis for the design of a computational model. Our controller extends the coupled dynamical systems framework and provides fast and synchronous control of the eyes, the arm and the hand within a single and compact framework, mimicking similar control system found in humans. We validate our model for visuomotor control of a humanoid robot

Modulating Vision with Motor Plans: A Biologically-inspired Efficient Allocation of Visual Resources

Abstract—This paper presents a novel, biologically-inspired, approach for an efficient management of computational resources for visual processing. In particular, we modulate a visual “attentional landscape ” with the motor plans of a robot. The attentional landscape is a more recent, general and a more complex concept of an arrangement of spatial attention than a simple ‘‘attentional spotlight ” or a ‘‘zoom-lens ” model of attention. A higher attention priority for visual processing must be given to manipulation-relevant parts of the visual field, in contrast with other, manipulation-irrelevant, parts. Hence, in our model visual attention is not exclusively defined in terms of visual saliency in color, texture or intensity cues, it is rather modulated by motor (manipulation) programs. This computational model is supported by recent experimental findings in visual neuroscience and physiology. We show how this approach can be used to efficiently distribute limited computational resources devoted to visual processing, which is very often the computational bottleneck in a robot system. The model offers a view on the well-know concept of visual saliency that has not been tackled so far, thus this approach can offer interesting alternative prospects not only for robotics, but also for computer vision, physiology and neuroscience. The proposed model is validated in a series of experiments conducted with the iCub robot, both using the simulator and with the real robot. I

Motor-Primed Visual Attention for Humanoid Robots

We present a novel, biologically inspired, approach to an efficient allocation of visual resources for humanoid robots in a form of a motor-primed visual attentional landscape. The attentional landscape is a more general, dynamic and a more complex concept of an arrangement of spatial attention than the popular "attentional spotlight" or "zoom-lens" models of attention. Motor-priming of attention is a mechanism for prioritizing visual processing to motor-relevant parts of the visual field, in contrast to other, motor-irrelevant, parts. In particular, we present two techniques for constructing a visual "attentional landscape". The first, more general, technique, is to devote visual attention to the reachable space of a robot (peripersonal space-primed attention). The second, more specialized, technique is to allocate visual attention with respect to motor plans of the robot (motor plans-primed attention). Hence, in our model, visual attention is not exclusively defined in terms of visual saliency in color, texture or intensity cues, it is rather modulated by motor information. This computational model is inspired by recent findings in visual neuroscience and psychology. In addition to two approaches to constructing the attentional landscape, we present two methods for using the attentional landscape for driving visual processing. We show that motor-priming of visual attention can be used to very efficiently distribute limited computational resources devoted to the visual processing. The proposed model is validated in a series of experiments conducted with the iCub robot, both using the simulator and the real robot

IL-33/ST2 pathway drives regulatory T cell dependent suppression of liver damage upon cytomegalovirus infection

© 2017 Popovic et al. Regulatory T (Treg) cells dampen an exaggerated immune response to viral infections in order to avoid immunopathology. Cytomegaloviruses (CMVs) are herpesviruses usually causing asymptomatic infection in immunocompetent hosts and induce strong cellular immunity which provides protection against CMV disease. It remains unclear how these persistent viruses manage to avoid induction of immunopathology not only during the acute infection but also during life-long persistence and virus reactivation. This may be due to numerous viral immunoevasion strategies used to specifically modulate immune responses but also induction of Treg cells by CMV infection. Here we demonstrate that liver Treg cells are strongly induced in mice infected with murine CMV (MCMV). The depletion of Treg cells results in severe hepatitis and liver damage without alterations in the virus load. Moreover, liver Treg cells show a high expression of ST2, a cellular receptor for tissue alarmin IL-33, which is strongly upregulated in the liver of infected mice. We demonstrated that IL-33 signaling is crucial for Treg cell accumulation after MCMV infection and ST2-deficient mice show a more pronounced liver pathology and higher mortality compared to infected control mice. These results illustrate the importance of IL-33 in the suppressive function of liver Treg cells during CMV infection

Identification of the cell type that expresses IL-33 in inflammatory foci in infected liver tissue as F4/80⁺ macrophages.

<p>BALB/c mice were injected i.v. with 5x10⁵ PFU of WT MCMV (MW97.01) and liver tissue was harvested on day 5 p.i. (<b>A</b>) Consecutive serial 1-μm sections of liver tissue focusing on an infected hepatocyte (Hc) that is delimited from uninfected tissue by a sheath made up by a mononuclear cell infiltrate. The expression of the indicated marker molecules was tested in a two-color IHC (2C-IHC) staining. (<b>a</b>-<b>d</b>) Identification of the infected Hc by red staining of the intranuclear viral IE1 protein. (<b>a</b>) Focus-forming mononuclear cells are not CD31^<b>+</b> black-stained endothelial cells (EC). (<b>b</b>) Focus-forming mononuclear cells are not CD3ε^<b>+</b> black-stained cells, thus excluding α/ß and γ/δ T cells as well as NKT cells. (<b>c</b>) Identification of focus-forming mononuclear cells as black-stained F4/80^<b>+</b> macrophages (Mø). (<b>d</b>) IL33-expressing cells stained in turquoise-green color colocalize with focus-forming F4/80 macrophages in the neighboring section of image <b>c</b>. Counterstaining with hematoxylin. Arrows point to the indicated cell types exemplarily. The bar markers represent 50 μm throughout. (<b>B</b>) 2C-IHC verifying colocalization of F4/80 and IL33 on the cellular level. (<b>a</b>) Higher resolution image of an advanced, aged focus consisting of a cluster of dual-stained F4/80⁺ (red) IL33⁺ (turquoise-green) macrophages (Mø) surrounding an infected hepatocyte (Hc) that is identified by an intranuclear inclusion body. Note that dually-expressing macrophages localize also to liver tissue outside of a focus. (<b>b</b>) A young focus in which dual-stained F4/80^<b>+</b> (red) IL33⁺ (turquoise-green) macrophages cling to an infected hepatocyte (Hc) that shows the pathocytomorphology of an owl’s eye cell with an intranuclear inclusion body that indicates the late phase (L phase) in the viral gene expression program. Counterstaining with hematoxylin. Arrows point to sites of interest. The bar markers represent 50 μm.</p

Treg cells show an activated phenotype after MCMV infection.

<p>BALB/c mice were i.v. injected with 2x10⁵ PFU of WT MCMV (clone MW97.01) or left uninfected. (<b>A</b>) Absolute number of Treg cells in spleen and liver is shown. (<b>B</b>) Representative FACS plots and (<b>C</b>) graphs showing percentages and (<b>D</b>) median fluorescence intensity (MFI) of Ki-67 expression by naive Treg cells. (<b>F</b>) Bcl-2 expression by naive Treg cells. (<b>E</b>) Mice were treated with BrdU in drinking water for 6 days starting at the day of infection. Percentage of BrdU positive Treg cells on day 7 was determined. (<b>G</b>) Histograms show a representative expression of different markers by Treg cells from uninfected and 7 days infected mice. (<b>H</b>) Representative FACS plots and (<b>I</b>) graphs showing percentages and (<b>J</b>) median fluorescence intensity (MFI) of ST2 expression by Treg cells isolated from the spleen and liver of naive BALB/c and ST2^-/- mice. Data are shown as mean ± SEM of n = 3–5 mice from one representative experiment out of three. *p<0.05; ** p<0.01; ***p<0.001 from two tailed, unpaired Student’s t-test.</p

ST2^-/- mice show increased liver damage and mortality rate after MCMV infection.

<p>(<b>A</b>) BALB/c and ST2^-/- mice were i.v. injected with 2x10⁵ PFU of WT MCMV (MW97.01) and lymphocytes from spleen and liver were analyzed on day 7 p.i. Absolute number of Treg cells is shown. (<b>B</b>-<b>E</b>) BALB/c and ST2^-/- mice were i.v. injected with 10⁶ PFU of WT MCMV (pSM3fr-MCK-2fl clone 3.3) and analyzed on day 5 p.i. (<b>B</b>) AST and ALT serum levels were determined. (<b>C</b>) Scores of cumulative liver pathology for apoptosis, intranuclear inclusion bodies (INIBs), inflammation, and necrosis. Bars correspond to the mean score for each parameter. The height of each bar represents the mean of the total histological score (out of 12). (<b>D</b>) Representative H&E and (<b>E</b>) Caspase-3 staining of paraffin embedded liver sections. (<b>F</b>) BALB/c and ST2^-/- mice were i.p. injected with indicated doses of SGV MCMV. Survival rates were monitored daily. Data are shown as mean ± SEM of n = 3–5 mice from one representative experiment out of three. For survival monitoring n = 7. *p <0.05 and **p<0.01 from two tailed, unpaired Student’s t-test.</p